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Abstract. This paper describes the overlay protocol Chord using the formal- 
ism of Abstract State Machines. The formalization concerns Chord actions 
that maintain ring topology and manipulate distributed keys. We define a 
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1. Introduction 

A decentralized Peer-to-Peer system (P2P) [23] involves many peers (nodes) 
which execute the same software, participate in the system having equal rights and 
might join or leave the system continuously. In such a framework processes are 
dynamically distributed to peers, with no centralized control. P2P systems have no 
inherent bottlenecks and can potentially scale very well. Moreover, since there are 
no dedicated nodes critical for systems' functioning, those systems are resilient to 
failures, attacks, etc. The main applications of P2P-systems involve: file sharing, 
redundant storage, real-time media streaming, etc. 

P2P systems are frequently implemented in a form of overlay networks [24j . a 
structure that is totally independent of the underlying network that is actually 
connecting devices. Overlay network represents a logical look on organization of 
the resources. Some of the overlay networks are realized in the form of Distributed 
Hash Tables (DHT) that provide a lookup service similar to a hash table; (key, 
value) pairs are stored in a DHT, and any participating peer can efficiently retrieve 
the value associated with a given key. Responsibility for maintaining the mapping 
from keys to values is distributed among the peers, in such a way that any change 
in the set of participants causes a minimal amount of disruption. It allows a DHT 
to scale to extremely large number of peers and to handle continual node arrivals, 
departures, and failures. The Chord protocol [201 - 122] is one of the first, simplest 
and most popular DHTs. The paper JO] which introduces Chord has been recently 
awarded the SIGCOMM 2011 Test-of-Time Award. 

Our aim is to describe Chord using Abstract State Machine (ASM) [11] and 
to prove the correctness of the formalization, which was motivated by the obvious 
fact that errors in concurrent systems are difficult to reproduce and find merely by 
program testing. There are at least two reasons for using ASMs. First, ASMs are 
versatile machines which are able to simulate arbitrary algorithms in a direct and 
essentially coding-free way. Here the term algorithm is taken in a broad sense in- 
cluding programming languages, architectures, distributed and real-time protocols, 
etc. The simulator is not supposed to implement the algorithm on a lower abstrac- 
tion level; the simulation should be performed on the natural abstraction level of 
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the algorithm. Second, a vast literature on ASMs shows how to model closely and 
faithfully real complex systems and how to use models in order to verify their prop- 
erties (see for example [5], Bakery algorithm [4 , Rail road crossing problem |13j . 
Kerberos algorithm [3 , Java formalization [7], jS], a special issue devoted to the 
method [6] , etc) . The ASM-code for Chord presented in this paper has been written 
following one of the best implementations [10] of the high level C++-like pseudo 
code from [22]. Actually, the main difference between the implementation [10] and 
our specification concerns improving efficiency of detection of a failed successor 
proposed in [35] which we have added to the code. 

Recently, several non-relational database systems (NRDBMS) have been devel- 
oped 9 , 16 that are usually based on the Chord like technology. To analyze their 
behavior, it might be useful to characterize situations when correctness of the under- 
lying protocol holds. Following that idea, we have formulated several deterministic 
conditions that guarantee correctness of Chord, and proved the corresponding state- 
ments. This is in contrast to the approach from 17, 20 -22] where a probabilistic 
analysis is proposed, and correctness holds with "high probability". 

The main objectives of Chord are maintaining the ring topology as nodes con- 
currently join and leave a network, mapping keys onto nodes and distributed data 
handling. The formalism of ASM enables us to precisely describe a class of possible 
runs - so called regular runs - of the protocol, and to prove correctness of the main 
operations with respect to it. Moreover, several examples of runs, given in Example 
15.11 that violate the constraints for the regular runs illustrate how correctness can 
be broken in those cases. 

We are aware of only a few attempts to formally verify behavior of DHTs and 
particularly Chord [ni2l lTlfllTTll2"5H2"7] . We consider them below and compare with 
our approach. 

The rest of the paper is organized in the following way. Section [2] introduces the 
notion of ASM. Chord is informally described in Section[3l Our ASM formalization 
of Chord is given in Section |4j where we present the general program and a high 
level description of the rules executed by every peer. In Section[5jthe class of regular 
runs is defined, and statements and the corresponding proofs of the correctness of 
the formalization are given. Section [6] presents discussion and comparison with 
related works. We conclude in Section [7] In Appendix [A] we provide a detailed 
specification of the rules involved in our formalization of Chord. 

2. Abstract State Machine 

We assume that the reader is familiar with the semantics of the Abstract State 
Machine defined in [5j[TTl[T2] , and we quote here only the essential definitions. 

A Gurevich's Abstract State Machine A is defined by a program Prog - consisting 
of a finite number of transition rules, at most countable set of states and initial 
states. A models the operational behavior of a real dynamic system S in terms of 
evolution of states. 

A state S is a first-order structure over a fixed signature (which is also the 
signature of A), representing the instantaneous configuration of S. The value of a 
term t at S is denoted by [t]s- The basic transition rule is the following function 
update 

f(ti, . . . , t n ) :=t 
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where / is an arbitrary n-ary function and t\, . . . , t n , t are first-order terms. To fire 
this rule in a state S evaluate all terms ti, . . . , t n , t at S and update the function / to 
[t]s on parameters [ti]s, • • • , [t n ]s- This produces another state S' which differs from 
S only in the new interpretation of the function / (since states represent memory, 
function update represents change in the content of one memory location) . 
Additionally, we have the following transition rules. 

The conditional constructor produces "guarded" transition rules of the form: 

if g then 
Ri 

else 

R 2 
endif 

where g is a ground term (the guard of the rule) and R\, R2 are transition rules. 
To fire that new rule in a state S evaluate the guard; if it is true, then execute R\, 
otherwise execute i?2- The else part may be omitted. 

The seq constructor produces transition rules of the form: 

seq 
Ri 



endseq 

to apply R\ , . . . , R n sequentially. Note that the se^constructor is originally defined 
without the endseq-line, but we add it to improve readability. 
The par constructor produces transition rules of the form: 

par 
Ri 



endpar 

to apply R\ , . . . , R n simultaneously, when possible, otherwise do nothing. 

If U is a universe name, v is a variable, g(v) is a Boolean term and R is a rule 
then the following expression (choose constructor) is a rule with the main existential 
variable v that ranges over U and body R: 

choose v in U satisfying g(v) 
R 

endchoose 

If there is an element a S U such that condition g(a) is true, fire rule R (with a 
substituted for v), otherwise do nothing. 

To express the simultaneous execution of a rule R for each x satisfying a given 
condition ip (forall constructor): 

forall x with do 
R 

endforall 
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A run (or computation) of A is a finite or infinite sequence So; Si; £2; ■ • ■ where 
So is an initial state and every Si+i is obtained from Si executing a transition rule. 

In general runs may be affected by the environment. Environment manifests 
itself via so-called external functions. Every external function can be understand 
as a (dynamic) oracle. The ASM provides the arguments and the oracle gives the 
result. 

In a distributed Gurevich's Abstract State Machine A multiple autonomous 
agents cooperatively model a concurrent computation of S. Each agent a executes 
its own single- agent program Prog(a) as specified by the module associated with a 
by the function Mod. More precisely, an agent a has a partial view View(a; S) of 
a given global state S as defined by its sub-vocabulary Fun(a) (i.e. the function 
names occurring in Prog(a)) and it can make a move at S by firing Prog(a) at 
View(a; S) and changing S accordingly. The underlying semantic model ensures 
that the order in which the agents of A perform their actions is always such that no 
conflicts between the update sets computed for distinct agents can arise. The global 
program Prog is the union of all single-agent programs. Nullary function Me, that 
allows an agent to identify itself among other agents, is interpreted as a for each 
agent a, and does not belong to Fun(a) for any agent a. It cannot be the subject 
of an update instruction and is used to parameterize the agent's specific functions. 
A sequential run of a distributed Gurevich's Abstract State machine A is a (finite 
or infinite) sequence So; Si; . . . ; S n ; . . . of states of A, where So is an initial state 
and every S ra +i is obtained from S n by executing a move of an agent. The partially 
ordered run, defined in (llj , is the most general definition of runs for a distributed 
ASM. In order to prove properties on a partially ordered run, the attention may 
be restricted to a linearization of it, which is, in turn, a sequential run (see [llj for 
more explanations). In the rest of the paper we consider only regular runs in which 
a state is global and moves of agents are atomic. 

3. (Informal) Description of Chord 

We assume that a number of nodes running the Chord protocol form a ring- 
shaped network. The main operation supported by Chord is: 

• mapping the given key onto a node using consistent hashing. 
The consistent hashing |14) provides load-balancing, i.e., every node receives roughly 
the same number of keys, and only a few keys are required to be moved when nodes 
join and leave the network. Chord networks are overlay systems. Thus, each node 
in a network (with A-nodes) needs "routing" information about only a few other 
nodes (O(logiV)), and resolves all lookups via O(logiV) messages to other nodes. 
When the network is not stable, i.e., the corresponding "routing" information is out 
of date since nodes join and leave arbitrarily, the performance degrades. However, 
a recent development |25j has shown that Chord's stabilization algorithm (with mi- 
nor modifications) maintains good lookup performance despite continuous failure 
and joining of nodes. 

Identifiers are assigned to nodes and keys by the consistent hash function. The 
identifier for a node or a key, hash(node) or hash(key), is produced by hashing IP 
of the node, or the value of the key. The length of identifiers (for example m bits) 
must guarantee that the probability that two objects of the same type are assigned 
same identifiers is negligible. Identifiers are ordered in an identifier circle modulo 
2 m . Then, the key k is assigned to the node such that hash(node) = hash(key). If 
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such a node does not exist, the key is assigned to the first node in the circle whose 
identifier is greater than hash(key). 

Every node possesses information on its current successor and predecessor nodes 
in the identifier circle. To accelerate the lookup procedure, a node also maintains 
routing information in the form of the so-called Finger Table with up to m entries. 
The i th entry in the table at the node n contains the identifier of the first node s 
that succeeds n by at least 2 I_1 in the identifier circle, i.e., s = successor(n + 2 l ~ 1 ), 
where 1 ^ i ^ m (and all arithmetic is preformed modulo 2 m ). The stabilization 
procedure implemented by Chord must guarantee that each node's successor pointer 
and finger table are up to date. The procedure runs periodically in the background 
at each node. To increase robustness, each Chord node can create a successor list 
of size r, containing the node's first r successors. 

Beside the mapping of keys onto the set of nodes, the only other operations 
realized by Chord are: 

• adding/removing of a node to/from a network. 

When a node n joins an existing network, certain keys previously assigned to n's 
successor now become assigned to n. When node n leaves the network regularly, it 
notifies its predecessor and successor and reassigns all of its keys to the successor. 

4. ASM FORMALIZATION OF CHORD 

4.1. Basic Notions. Let L, M and K be three fixed positive integers, and TV = 
2 M . We will consider the following disjoint universes: 

• the set Peer = {pi, . . . ,p^} of all peers that might participate in the con- 
sidered Chord network, 

• the set Key — {k\, . . . , Uk } of identifiers of objects that might be stored 
in the considered Chord network, and the set Value = {v\, ...,Vk} of the 
values of those K objects, 

• the set Chord — {0,1, . . . ,N — 1} denoting at most N peers that are in- 
volved in the network in a particular moment, 

• the sets Join = {join, skip} and Action = {put, get, fair leave, unfair leave, skip} 
which represent the actions of the peers, 

• the sets Mode = {not -connected, connected}, Mode-join — {undef, wait-f or successor , 
wait-j 'or -keys, finished}, Modeleave = {undef , wait-f or successor , proceed-to-f inish} , 
Modestabilize = {undef, wait-f or successor, wait-f or -predecessor, wait-f or -keys, 
wait-f or successor -keys, finished}, Mode-get = {undef, wait-f or -key , wait-f or -value} , 
Mode-put = {undef , wait-f or -response}, Mode- fingers — {undef , wait-f or -response} 
which represent the states of the peers, 

• the set CommunicationType = {get, findsuccessor, sendsuccessor, send-predecessor, 
send-keys, set -predecessor, request-and-remove-keyvalue, successor-found, value-found, 
get -predecessor, received-keyvalue, received-predecessor} of special miliary 

functions denoting type of communication, 

• the set ContentType = {Chord x Value}* U {Chord x Chord} U Chord U 
Value which represents all possible types of messages content, and 

• the set Message — CommunicationType x ContentType which represents 
messages that are exchanged by the nodes. A messages is defined by its 
type and content. 

Note that: 



6 



BOJAN MARINKOVIC, PAOLA GLAVAN, AND ZORAN OGNJANOVIC 



= REjEi:ESSCs. 



SUCCESSOR 



FINGER TABLE 



Figure 1 . Structure of Chord node 



• it might be that L > N (K > N), i.e., that there are more peers (objects 
to be stored in the network) than nodes, but it can never be N > L, and 

• without any loss of generality we assume that the numbers of keys and 
values are the same; if there are more values than keys, all values mapped 
to the same key might be organized in a list. 

In the sequel, we will use the following (standard) list-manipulation functions: 

• list constructor, 

• add a new element to the list, 

• remove an element from the list, and 

• listitem returns the i th element of a list, 

and the strict and the total orders of N (denoted by <, and ^). Also, we will use 
a ©jv b to denote (a + b) mod N. 

Any peer, active in the network will be called a node. We assume that a 
node (Fig. [TJ is represented by its identifier id in the network, information on 
its predecessor and successor, a finger table, a pointer (next) to an element in 
the finger table which will be updated in the current stabilization cycle, and a list 
of {key, value) pairs of the records for which the node is responsible for. 

More formally, we introduce the following functions: 

• id : Peer — > Chord U {undef} 

• successor : Chord — > Chord, 

• predecessor : Chord — > Chord, 

• finger : Chord — > Chord*, 

• next : Chord — > {1, . . . , M}, and 

• keyvalue : Chord — > (Chord x Value)*, 

where Chord* is the set that contains lists of nodes' identifiers, and (Chord x 
Value)* is the set of lists containing pairs (hash(key) , value) . Each finger(x) has 
M entries ordered respect to the ring ordering. 

In other words, a peer p, which is a node, is represented by the tuple: 

• (id(p), successor (id(p)) , predecessor (id(p)) , finger(id(p)), next(id(p)), keyvalue(id(p))) . 

Table [T] shows all the other functions that will be used in the formal description 
of the protocol, but that do not directly change the representations of nodes. We 
assume that the five functions in Table[T](/ias/i, ping, known-nodes, key -value and 
keys) are external. 
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Function 


Description 


hash 


Maps the sets of peers and keys to Chord U {undef} 


vino 


Tests whether a node is reachable 


member -of 


Checks whether a node is between two nodes in Chord 


communication 


Realizes communication requests 


mode 


Determines Peer-agent state 


mode-join 


Determines Peer-agent state during join operation 


modeJeave 


Determines Peer-agent state during fair leave operation 


mode stabilize 


Determines Peer-agent state during stabilize operation 


mode-fingers 


Determines Peer-agent state during update of finger table member 


modejput 


Determines Peer -agent state during put operation 


mode-get 


Determines Peer -agent state during get operation 


known-nodes 


Simulates external knowledge about existing nodes 


key_value 


Select a (key, value) for storing in the network 


keys 


Select to look for a value with particular key 



Table 1. Chord functions 



The hash function assigns identifiers of nodes to peers and keys: 

• hash : Peer U Key — > Chord U {undef}, 
where undef is a special value which indicates that: 

• there are N nodes in the network, and an identifier is requested for a new 
node, or 

• there are N keys in the network, but we try to add a new key. 

The function must also guarantee that in each moment two different active peers 
(keys) have different hash values. However, note that it is possible that in different 
moments different peers have the same identifier. Also, it may happen that a 
peer can have different identifiers (obtained by different calls of the hash function 
before and after a period in which the peer is not present in the network). The 
above mentioned id function can be explained as a "local" counterpart of hash. 
Namely, we can assume that the values produced by hash are stored in the local 
memory and read and published by id to reduce the number of expensive calls of 
hash. In the program given below, id will be invoked with the argument Me to 
allow a node to identify itself in the network. 
The external function ping, defined as: 

• ping : Chord — > {true, false} 

returns true or false, depending on whether the argument is reachable in the 
network. 

The function member-of. 

• member-of : Chord x Chord x Chord — > {true, false}, 

determines whether the first argument is between two next two arguments with 
respect to the ring ordering, more formally: 

• if argi = arg$ always returns true, 

• if arg2 < arg% returns true if argi < arg± ^ arg% holds, 

• if argi > arg% returns true if ~^(arg^ ^ arg\ < argi) holds, 

• otherwise returns false. 
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Figure 2. member -of 



Example 4.1. Let N = 64. Then, 

• member_of(3l, 14, 14) = true, see Fig. [2ja, 

• member jof{3>\, 14,48) = true, see Fig. [3Jb, 

• member. of (62, 14,48) = false, see Fig. [5]b, 

• member _of (31,48, 14) = false, see Fig. [2]c, 

• member _of (62,48, 14) = irwe, see Fig. [2jc. 



In this paper we will realize communication following the ideas from [3] and [5]- 
During communication a direct channel is open between two nodes (assuming that 
channels do not loose information): 

• communication : Chord x Chord — > Message* . 

The first argument is the sender and the second one is the receiver of a message. 
When a sender sends a new message to a receiver, it appends it to the list of 
all messages which where sent by this sender to that receiver. When the receiver 
processes this message, it removes it from the list. Different type of messages 
contain different information: 

• if the type of communication is request_andjremove-keyvalue, the message 
is empty, 

• if the type of communication is sendJteys, the message contains all keyvalue- 
pairs that belong to the sender 

• if the type of communication is set-predecessor, the message is empty 

• if the type of communication is getjpredecessor, the message contains 
predecessor of the sender 

• if the type of communication is find successor, the message contains two 
values - the value for which the successor is asked, and the id of the peer 
which initiated the query 

• if the type of communication is get, the message contains the hashed value 
of the key 

• if the type of communication is successor -f ound, the message contains the 
value for which successor was asked and information about the successor 

• if the type of communication is value-found, received-key value or received-predecessor , 
the message contains the resulting information. 

The function mode: 



□ 
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• mode : Peer —> Mode 

determines Peer-agent state. Initially, for all p 6 Peer value of mode{p) is set to 
not -connected. 

The functions mode-join, modeJeave, modestabilize, mode-fingers, modejput 
and mode-get: 

• mode-join : Peer — ► Mode-join 

• modeJeave : Peer — > ModeJeave 

• modestabilize : Peer — > M odestabilize 

• mode-fingers : Peer — > Mode-fingers 

• modejput : Peer — > Mode-put 

• mode-get : Peer — > Mode-get 

determine Peer-agent states during some of the Chord operations. Initially, for all 
p G Peer the values of these functions are set to undef . 
The external function known-nodes: 

• known-nodes : Peer —> Chord 

simulates external knowledge about the nodes in the particular Chord network. 
The external function keyjualue: 

• keyjualue : Peer — ¥ Key x Value 

simulates the choice of a node to store a (key, value) pair in the Chord network. 
The external function keys: 

• keys : Peer — > Key 

simulates the choice of a node to look if some value with particular key is stored 
in the Chord network. 

4.2. Chord Rules. The rest of this section contains our ASM-formalization of the 
Chord protocol. We present the general program executed by every peer, and a high 
level description of the rules performed in a Chord network which corresponds to 
the pseudo code given in [22] (note that the rules FairLeave, UnfairLeave, Put 
and Get are not given there). A detailed specification of these rules is provided in 
Appendix [A] 

4.2.1. Peer-agent Module. The following main module contains actions that are 
executed by every peer. The mode of all peers is initially not-connected. After 
a node joins a network successfully, its mode is changed to connected. In each 
execution of a loop, a node concurrently calls the rules responsible for the ring 
topology maintenance (Stabilize, UpdatePredecessor, UpdateFingers) and 
communication (ReadMessages) and, according to a non-deterministic choice, it 
might also invoke one of the FairLeave, UnfairLeave, Put and Get rules. 

if mode(Me) = not -connected then 
if Choosed Action Is Join 
seq 

if There Are No Known Nodes then 

Start 
else 

Join 
endif 

if Connection Successful then 
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mode(Me) :— connected 
else 

mode(Me) :— notjzonnected 
endif 
endseq 
endif 
else 

if mode(Me) = connected then 

if id(Me) Does Not Have Communication Problems then 
par 

ReadMessages 
Stabilize 

UpdatePredecessor 
UpdateFingers 
Extended J oinM odel = 
seq 

choose action in Action 
par 

Leaving Actions = 

FairLeave Or UnfairLeave 
K eyV alueH andling = 
Put Or Get 
endpar 
endseq 
endpar 
else 

mode(M e) := not ..connected 
endif 
endif 
endif 

4.2.2. Chord Rules - High Level Description. In the sequel, we will describe the 
rules of the Chord protocol (listed in Table [2]). 

When Nodei starts a new Chord network, the following rule is executed: 

Start= 
seq 

id(Me) := hash(Me) 

Initialize Values For Node id(Me) 
endseq 

When Nodei asks Nodej, which is the member of a Chord network, to join the 
network the following rule is executed (a network is fulfilled if it already contains 
N nodes): 

JOIN= 

seq 

id(Me) := hash(Me) 
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Rule 


Description 


Resulting State 




J_ lll_< ill o L IIULIL o bcLL LB Lll^ LWU1J\ 


r\ BlalL WlLll UllL IIULIL 


Join 


A tipcat nnnp iaimc f hp tip1~'wp*y - k' 

1 \ 11LW 11UUL J^Jlllo LllL llv_ L v V vJI IV 


A ^tatp with £in additional 

■li. OldUL W 1 U11 dill Cl*_-L*_ll UlUllCLl 

node 


FairLeave 


A node leaves fairly the network 


A state without one node 


T Tisitta trT ,~P A VP 


A n r\ r\ t~\ \Cid\TC*c /pv'ion^c 


A c t a t p i*7"i t n p»i 1 1 on p n p»d p 


Stabilize 


A node updates its successor 


Successor and predecessor 




and nred eres^or 


nnda to 


UpdatePredecessor 


Periodic check of the predecessor 


Predecessor update 


UpdateFingers 


A node runs update on its 
finger table 


Updating finger table entries 


Put 


A new (key, value) pair is stored 


Updating (key, value) table 


Get 


Finding a value for a given key 


Unchanged state 


FindSuccessor 


Finding a responsible node for 
given key or successor of a node 


Unchanged state 


ReadMessage 


Read messages dedicated to a node 


Changing some local 
variables if it is requested 



Table 2. Chord rules 



if Chord Is Not Fulfilled then 
seq 

Initialize Values For Node id(Me) With Respect To known 
Take keyvalue That Should Belong To 
Node id(Me) From successor (id(Me)) 
endseq 
endif 
endseq 

After an application of the join rule, all keys which have the hash value less or 
equal then the id of the new node, and which have been stored in the keyvalue list 
of the successor of the new node are moved to the keyvalue list of the new node. 

When Nodei leaves a Chord network in a fair way, it executes the following rule: 

FairLeave= 
seq 

if successor (id(Me)) Is Not Alive then Update successor(id(Me)) 
endif 

if id(Me) Is Not The Only Node In Chord then 
seq 
par 

Give keyvalue(id(Me)) To successor (id(Me))) 
Remove id(Me) From Successor Pointers Ring 
endpar 

Deactivate Values For Node id(Me) 
endseq 
endif 
endseq 
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Note that a node can also leave a Chord network in an unfair way, for example 
caused by a node crash, communication problems, etc. As it will be shown in Section 
[5] the other rules can detect such situations. Thus, the body of UnfairLeave rule 
is empty. 

The next three rules are responsible for detecting situations in which the consid- 
ered Chord network is not in a stable state, updating information about the network 
and reconnecting the successor pointers of nodes to establish a stable state. These 
rules are applied periodically, while a peer belongs to the considered Chord network: 

Stabilize= 

if successor (id(Me)) Is Alive then 
seq 

Set x To Be predecessor(successor(id(Me))) 
if (x^undefAmember_of(x,id(Me),successor(id(Me)))) then 
seq 

successor (id(M e)) := x 

Take Keyvalue That Should Belong To 
Node id(Me) From successor (id(Me)) 
endseq 
endif 

if x = undefV(x ^ undef Nmemberjof (id(M e) , x, successor(id(M e)))) then 

Notify successor (id(Me)) To Update Its predecessor To id(Me) 
endif 
endseq 
else 

Update successor (id(Me)) 
endif 

Stabilize is responsible to update information on successor of a node either 
because the successor is crashed or the new node with appropriate id has enter the 
network, and to notify new successor about its new predecessor. 

Checking if the current predecessor of a node is still active is realized by: 

UpdatePredecessor= 

if predecessor (id(Me)) Is Not Alive then 

Deactivate Values For predecessor (id(Me)) 
endif 

Updating the values from finger table is realized by firing: 
UpdateFingers= 

For Next next{id{Me)) Update finger.listitem(next(id(Me))) 
With Responsible Node For id(Me) ® N 2 next ^ Me ^- 1 

During one cycle of Peer_agent Module exactly one value from finger table is up- 
dated. 

Storing a new {key, value) pair is realized by the following rule: 
Put= 

if Chord Is Not Fulfilled then 

Notify Responsible Node For hash(key) To Add (hash(key), value) 
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Figure 3. findsuccessor 



To Its keyvalue Table 
endif 

The name of the rule FindSuccessor comes from [20]. By this rule a node is 
asked to return successor of the given argument. The corresponding argument can 
be the hash value of a key, or the id of a peer. In the former case, the result is the 
identifier of the member of the network which is responsible for the (key , value) 
pair. In the later case, the rule gives the identifier of the first member of the network 
which is equal to, or greater then, the argument. Note that, if the argument is the 
id of a node (i.e., a peer that is active in a network), the result is id, and not the 
identifier of its successor. 

FlNDSUCCESOR= 
For Given key 

if member _o/ (key, id(Me), successor ■(id(Me))) then 
Respond With successor (id(Me)) 

else 

Forward Query To Closes Predecessor From finger(id(Me)) 
endif 

In the above rule, for the argument h, the current node n returns its successor 
to the node which was started the query, if member -of(h, n, successor(n)) = true. 
Otherwise, the query is forwarded to the node whose id precedes h in n's finger 
table, or is the maximal element of this table. 

Example 4.2. Let N = 64 and a Chord network contain the nodes {1, 14, 28, 31, 32, 48} 
(see Fig. [3]) . Let the corresponding successor pointers form a ring. Note that there 
is no node with the identifier 29 in the network. The result of call of the successor 
of 29 by the node 14 is 31, because node 29 is understood as the hash of a key or 
the id of a node entering the network. The node 14 will transfer the query to its 
successor (node 28). This node will return its successor (node 31) as the result, 
because member .of (29,28,31) is true. 

However, since the node with the identifier 31 exists in the network, the result 
of the call of the successor of 31 will be 31. 

Finally, note that, if one needs information on the successor of a node, for ex- 
ample the node 31, then it should call for a successor of 31 ©64 1. □ 
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Any node present in a Chord network can execute Get rule (ask for the value 
of a key). That rule does not change the actual state of the network, but we define 
it as: 

Get= 

Invoke FindSuccessor For Given key 
And Check Corresponding value 

During the each execution of a Peerjagent Module all the messages send to a 
node are processed: 

ReadMessages= 

Read Messages Dedicated To Me , 

Change Local Variables If It Is Requested And 

Clear Processed Messages 

5. Correctness of the Formalization 

In this section we present the correctness of our formalization with respect to 
the so-called regular runs. First, following [llj we formally consider a distributive 
algebra A which consists of the module Peerjagent introduced in Section 14.21 a 
vocabulary T, and a collection of T-states, such that: 

• the vocabulary T contains: 

— all function names that appear in the module Peerjagent, except Me, 

— a miliary function name undef, the standard Boolean constants and 
operations, and 

— a unary function name Mod, 

• a state S of the vocabulary T is a pair which consists of: 

— a base set Peer U Key U Value U Chord U Action U Communication U 
{Peer _agent}, where Peerjagent is a module name, and 

— an T-interpretation I which satisfies that I(Mod) : Peer — > {Peerjagent} , 
while the other function names are interpreted as the corresponding 
objects defined at the beginning of Section SI 

• in a local state (of the vocabulary T U {Me}) which corresponds to the 
peer p £ Peer and the state S, denoted by View(p; S), all symbols are 
interpreted as in S, and additionally I(Me) = p. 

In the initial state a value mode(p) is set to not ..connected and id(p) is set to 
undef for all p € Peer, while values of successor(c), predecessor (c), finger(c), 
next(c) and keyvalue{c) are set to undef for all c 6 Chord. 

Definition 5.1. Let X\,X2 £ Node and yo, . . .y r € Node be all the nodes from a 
Chord network such that xi = yo < • • • < yr = X2- The pair {x\, X2) forms a stable 
pair in a state if the following holds: 

• yi+\ — successor (yA, — predecessor '(j/i+i) , for all i € {0, . . . , 1 — I}. 

A Chord network {xo, . . . ,Xk-i}, k 1, is stable in a state if the pair (xq,xq) is 
stable. □ 

Intuitively, a pair (xx,X2) is stable in a state if there is no node trying to join 
the network through the node on the ring-interval (£1,22) in that state. 
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A move at a state S is an execution of the module Peerjigent by a peer p at 
View(p; S). The corresponding rules belong to the low-level description given in 
Appendix [SJ Similarly as in we consider only atomic moves. 

A rurQ of our distributive algebra A is a triple {Moves, P, a) such that: 

• Moves is a partially ordered set of moves of peers such that every set 
{y : y ^ x} is finite, and y < x means that the move y must be finished 
before x begins, 

• the function P associates with every x 6 Moves a peer P(x) which executes 
x, and satisfies that each nonempty set {x : P(x) = p} is linearly ordered, 
and 

• the function a associates with every finite initial segment of Moves a state 
of A, while <t(0) denotes an initial state. 

Note that the assumption that every set {y : y ^ x} is finite implies the fairness, 
in the sense that every node will eventually execute its next move. Furthermore, 
we require that the following coherence condition holds for every run: 

• Let a; be a maximal element of a finite initial segment X of Moves, and 
Y = X\ {x}. Then, x transforms a(Y) to a(X). 

A state is reachable in a run (Moves, P, a) if it belongs to the range of a. 

A run p' = (Moves', P' , a') is an initial segment of a run p = (Moves, P, a) if 
Moves' is an initial segment of M oves and P' and a' are restrictions of P and a, 
respectively. A run p' is a linearization of a run p if Moves' is a linearization!! of 
Moves, P' and P coincide, and a' is a restriction of a. Note that each linearization 
is a sequential run. Two fundamental corollaries formulated in |1 1 j are: 

Corollary 5.1. (f) All linearizations of the same finite initial segment of a 
run have the same final state. 
(2) A property holds in every reachable state of a run p iff it holds in every 
reachable state of every linearization of p. 

Definition 5.2. Regular runs are all runs of a distributive algebra A which satisfy 
that: 

• any execution of FairLeave, UnfairLeave and Put might happen only 
between a stable pair of nodes. □ 

The following example illustrates the need for the above constraint. In the 
example and in the rest of the paper we will graphically illustrate sequences of 
moves, so that Si denotes a state, the updated values are in bold, and <0> means 
that the rest of a network is not affected by a move. 

Example 5.1. Let So be the initial state in which the nodes Ni and N3 are 
members of a network, and the node N 2 wants to join. Suppose that before the 
pair (Ni,N 3 ) becomes stable, Ni executes the put rule with the hash 2 of a key. 
Since Ni is not aware of N 2 , the corresponding key will be stored in N3, and not 
in N 2 . 



Note that this is a simplification of the corresponding definition from which results from 
the fact that the set Peer is fixed at the beginning and cannot be changed during executions of 
Chord. 

^Moves' is linearly ordered, Moves' and Moves have the same elements, and if x < y in 
Moves, then x < y in Moves' . 
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So 






Si 


id 


1 
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JV 2 Join 
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id 
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Stabilize 
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hashikey) 
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Again, assume that So is the initial state and the network contains the nodes 
Ni, N 3 and N 4 . If the node N 2 executes the join rule, and before the pair (Ni,N4,) 
becomes stable, N3 wants to leave, N2 will be isolated from the rest of the network, 
and the other nodes will never be aware of it. 



So 


Ni Join 


Si 


id 


1 
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id 
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4 


predecessor 





1 


3 




predecessor 





undef 
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4 





AT 3 FairLeave 


S 2 




id 
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2 
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predecessor 





undef 


1 


successor 


4 


3 






A similar example can be given for UnfairLeave. 



□ 



In the sequel, we show that a stable pair of nodes in a Chord network, which 
executes a regular run, eventually becomes stable after adding/removing of a node 
between them (the theorems I5.H5.6I) . Corollary 15.31 formulates the corresponding 
statement for a stable network. Finally, we prove that the proposed key-handling 
correctly distributes keys and answers queries (Theorem 15 . 71 and Corollary 15. 4p . 

The first theorem expresses that the rule FindSuccessor will terminate in a 
finite number of steps. It corresponds to Theorem IV. 2 from [2QH22] . 
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Theorem 5.1. Let n S Chord be the node which fires the rule FindSuccessor 
for h e {0,1,..., JV — 1}. Let m! be the minimal element of Chord such that 
h ^ ml . If the pair (n,m') is stable in that state, the node n will get the result 
after a finite number of moves. 

Proof. Let the node n be asked for the successor of h: 

• if h is a member of the ring-interval (n, successor(n)], successor(n) as the 
result is returned; otherwise 

• a node fc (such that the ring-interval (fc, h] is the smallest subset of the 
ring-interval (n, h] for every node from the finger table of n) is chosen to 
answer the query. 

Then, FindSuccessor is invoked by the node fc. The number of such calls is 
limited since there are finitely many nodes between n and h. Furthermore, in 
each step the ring-interval (fc, h\ shrinks, and since - by the construction - the first 
element of the finger table of a node is its successor, we will eventually reach the 
node m such that h G (m, successor (m)}. Note that in each step a node (an active 
member of the network) is contacted, and that the result is directly sent to the 
node n which issued the query. Node n must be active at that moment. It cannot 
execute (Un)FairLeave before it gets the answer, since (Un)FairLeave actions 
can be executed only when nodes do not wait for answers from earlier fired queries. 

□ 



Theorems 15 . 2 [ - 15761 guarantee that the successor and predecessor pointers for each 
node will be eventually up to date after a node joins, or unfair leaves the network. 
In the corresponding proofs we will use some finite initial sequences of runs. Due 
to the fact that the Stabilize and UpdatePredecessor are applied periodically 
by all nodes in a network, we will mention only those applications which change 
the values of the functions predecessor and successor. 

Note that, in each proof we will consider some fixed linearization of moves, but 
according to Corollary 15 .![ all linearizations of the corresponding regular run will 
result in the same final state. 

Theorem 15.31 corresponds to Theorem IV. 3 from 



Theorem 5.2. Let a peer join a Chord network, between two nodes which consti- 
tute a stable pair. Then, there is a number fc > of steps, such that if no other 
join rule happens in the meantime, the Stabilize rule will bring the starting pair 
to be stable after fc steps. 

Proof. Suppose that the network contains only one node and that Ni 2 wants 
to join. We will consider the following sequence of moves: 



So 


id 


h 


predecessor 


h 


successor 


h 



N io Join 



Si 


id 






predecessor 


i\ 


undef 


successor 


h 





s 2 


id 


ii 


ii 


predecessor 




undef 


successor 


h 


k 



N io Stabilize 
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At this moment the pair is not stable, and the restriction to the regular runs does 
not allow the nodes to try to leave the network. Also, according to the assumption 
of the statement, other peers will not execute the Join rule. Then, we have: 



S 2 



JVij Stabilize 



S3 


id 


h 




predecessor 




undef 


successor 




ii 



Stabilize 



S4 


id 


h 




predecessor 




ii 


successor 


i2 


h 



Obviously, a stable pair has been established, again. 

Suppose that there are two or more nodes in the network, and that Ni k wants to 
join. Let Ni k _ 1 and Ni k+1 be the members of the network such that successor(ik-i) = 
ik+i, and predecessor(i k+ i) = i k -i (i.e., (JV jfc _ 1 , N ik +1 ) is a stable pair). Fur- 
thermore, let member jo f{ik,ik-\,ik+i) = true. Then, we consider the following 
sequence of moves: 



So 


id 


ik-i 


ik+1 


predecessor 





h-i 


successor 


ik+l 






Ni, Join 



Si 


id 


ik-i 


ik 


ik+l 


predecessor 





undef 


ik-i 


successor 


ik+l 


ik+l 








s 2 


N iu Stabilize 
> 


id 


ik-i 


ik 


ik+l 




predecessor 





undef 


Ik 




successor 


ik+i 


ik+l 






Similarly as above, the restriction to the regular runs docs not allow the nodes to 
try to leave the network, other peers will not execute the Join rule, and we have: 



Stabilize 



Ni, .Stabilize 

^ 



S3 


id 


ik-i 


ik 


ik+l 


predecessor 





undef 


ik 


successor 


ik 


ik+l 
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id 


ik-i 


Ik 


ik+l 


predecessor 





ik-i 


ik 


successor 


Ik 


ik+l 






Thus, a stable pair has been established. 



□ 



Theorem 5.3 (Concurrent joins). Let a Chord network contain a stable pair. If a 
sequence of Join rules is executed between the nodes which form this stable pair, 
interleaved with Stabilize, UpdatePredecessor and Update_fingers, then 
there is a number k > of steps, such that after the last Join rule, the starting 
pair of nodes will be stable after k steps. 

Proof. First, note that UpdateFingers does not change the values of the functions 
predecessor and successor. Similarly, UpdatePredecessor might change values 
of the function predecessor only after an UnfairLeave. Thus, we do not consider 
executions of UpdatePredecessor and UpdateFingers in the rest of this proof. 
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Let us assume that all peers that want to join the network have different suc- 
cessors. Then, by Theorem I5.2[ the statement holds. Otherwise, there must be 
at least two peers that want to join the network having the same successor. Sup- 
pose that Ni 2 and N{ 3 want to join and that and A^ 4 are members of the 
network, such that successor(ii) = ii and predecessor(ii) = i\. Furthermore, let 
member jof (12,11,13) = true and member _of (13,12,14) = true. Then, we consider 
the following sequence of moves: 



So 


id 


h 


u 


predecessor 





ii 


successor 


u 






iVi,JOIN 
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id 




h 


U 
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undef 
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S 2 




, Stabilize 
> 


id 


i\ 


12 


13 


ii 


Ni 
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successor 
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ii 
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Ni, Join 



At this moment, the successor pointers of and Ni 3 connect those nodes and 
iVj 4 . Then, similarly as in Theorem 15.21 executions of Stabilize by Ni 2 and 
result with the stable pair (Ni 1 , Ni 4 } in the state Sg : 
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The execution of the Stabilize rule by Ni 3 in S2 does not change the values 
predecessor^) and successor^)- The same holds if more than one node (denoted 
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Nj x , Nj 2 , . . . ) join the network between and N{ 3 . So, let us assume that 

i\ < • • • < h < ji < h < «4- 

Then, similarly as above, a sequence of executions of the Stabilize rule by the 
nodes Nj 1 , , Nj 2 , iV^ , . . . result with a stable starting pair. □ 



Theorem 5.4. Let a Chord network contain a stable pair and let a node between 
them leave the network. Then, there is a number k ^ of steps, such that if no 
Join rule happens at the considered part of the network in the meantime, the pair 
will be brought into a stable state after k steps. 



Proof. Let us first assume that the node leaves the network in a fair way. Since 
FairLeave produces a stable pair, the statement holds for k = 0. Thus, let 
UnfairLeave be executed. 

Suppose that the network contains only two nodes N il and N i2 , and that N i2 
leaves in an unfair way and breaks the ring. Then, we consider the following 
sequence of moves: 
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id 
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h 


successor 


h 



The state S4 is stable. 

Suppose that there are three or more nodes in a network. Let N ik _ 1 , N ik 
and Ni k+1 be the members of the network such that successor(ik-i) = ik and 
successor(ik) — ik+i- Suppose that Ni k unfair leaves and breaks the ring of the 
successors pointers. Then, we consider the following sequence of moves which re- 
sults with the stable pair {N ik _ 1 , N ik+1 ) in state S4: 
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s 3 
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Theorem 5.5. Let a Chord network contain a stable pair. Let a node which is 
between those nodes leave the network following by several nodes which want to 
join between them. Then, there is a number k ^ of steps, such that the considered 
pair will be brought into a stable state after k steps. 

Proof. Let us first assume that the node leaves the network in a fair way. It pro- 
duces a stable pair, and according to the theorems 15.31 and 15.41 the statement 
holds. Then, let the node Ni 2 execute UnfairLeave and break the ring. If no 
node joins the network in the ring interval [predecessor fo), successor^)], the 
statement holds similarly as in theorems 15.31 and 15.41 Finally, assume that a node 
joins the network in the ring interval [predecessor (12), successor^)]. Suppose 
that Ni t , Ni 2 and Ni t are members of the network, such that successor(ii) = 12, 
predecessor^) — i\, successor^) = ii and predecessor (14) = 12. Furthermore, 
let member s>f (i-i^iiii-i) = true and member_of(i 3 , 12,14) = true. Let N i2 be the 
node that will leave, and Ni 3 node that will join the network. We consider the 
following sequence of moves: 
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which results with the stable starting pair in the state <SV- If more than one node 
want to join the network in the considered ring interval, we can use similar argu- 
ments as in the proof of Theorem 15.31 to establish the statement. □ 

Note that the restriction from the formulation of Theorem 15.51 that no other 
leave-rules are allowed after the first one, is not essential. According to the definition 
of regular runs, leave-rules can be executed only between nodes which constitute a 
stable pair, and we can consider an execution of a sequence of join rules interleaved 
with leave-rules, and obtain the same result. The above statement will hold for each 
subsequence which starts with a leave rule followed by several join rules. Thus, we 
have the following: 

Corollary 5.2. Let a Chord network contain a stable pair. Let a node, which is 
in between those nodes, leave the network. Then, there is a number k ^ 0, such 
that the considered pair of nodes will become stable after k moves. 

Theorem 15.61 incorporates all previous ideas, and is the main statement concern- 
ing correctness of maintaining topological structure of Chord networks. 

Theorem 5.6. Let a finite initial segment of a run produce the state S of a Chord 
network. Then, for every pair of nodes n,n' G Chord, there is a number k 0, 
such that (n,n') will become stable after k moves. 

Proof. The case in which the state S is stable is trivial. So, let us assume that 
S is not stable. According to the definition of regular runs, no leave rule might 
happen before the network becomes stable. Thus, only a sequence of Join rules 
interleaved with Stabilize, UpdatePredecessor and UpdateFingers can be 
executed. Since the number of nodes in the network is limited, the number of 
join rules in the sequence must be finite. Then, similarly as in Theorem 15.51 the 
statement holds. □ 

Since a network is stable in a state if all pairs of nodes from the network are 
stable in that state, we have: 

Corollary 5.3. Let a finite initial segment of a run produce the state S of a Chord 
network. Then, there is a number k ^ 0, such that the network will become stable 
after k moves. 

Finally, the next two statements say that our formalization consistently manip- 
ulates distributed keys. Theorem 15.71 states that (key, value) pairs are properly 
distributed over the network. Informally, it follows from the facts that for every 
n G Chord, hash(key) ^ n for the keys for which n is responsible for, and that all 
rules that manipulate (key, value) pairs invoke FindSuccessor rule. 

Theorem 5.7 (Golden rule). 

\/((key, value) G Keys x Values, n G Chord) ((key, value) G keyvalue(n) 
member _of (hash(key) , predecessor (n) , n). 

Proof. Obviously, the statement holds for a Chord network containing only one 
node. According to the definition, a Join executed by Ni moves to Ni the cor- 
responding (key, value) pairs stored in its successor. Similarly, FairLeave and 
Stabilize executed by N transfer all (key, value) pairs from Ni to its successor. 
When a node leaves the network in a unfair way, all (key, value) pairs it stored 
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would be lost. Thus the statement remains true. By the definition, for a given 
(key, value) pair, Put finds a node responsible for the pair which satisfies the 
statement. Finally, all the other rules do not manipulate (key, value) pairs. □ 

Corollary 15 .41 follows from the definition of Get, and the theorems 15.11 and 15.71 

Corollary 5.4. If Get returns undef for some key e Keys, then there is no 
value € Values such that (key, value) pair is stored in the Chord network. 

Namely, according to Theorem l5.7[ all (key, value) pairs are stored properly, and 
from Theorem 15.11 Get considers only the (key, value) pairs stored in the node N 
which satisfy condition member _o/( hash(key),predecessor(id(N)),id(N)). 

6. Discussion and Related Works 

One of the basic ideas behind the results presented in Section [5] is that we 
consider not all possible, but only regular runs. Example 15.11 shows that without 
that restriction the ring topology can be damaged which might result in splitting 
a Chord network in disjoint parts and/or in loosing data. 

The request that every execution of the module Peer_agent by a peer is atomic 
can be relaxed so that only executions of the Chord-rules are required to be atomic. 
In that case it is necessary to introduce, so called, Time-To-Leave (TTL) mecha- 
nism. It means that if some rule does not finish in predefined number of module 
executions, a node repeats execution of that rule from the beginning. 

On the other hand, the atomicity assumption is essential and cannot be com- 
pletely avoided since it eliminates several observed shortages, for instance when a 
node tries to join the existing network via another node which tries to leave in the 
same moment. Another counterexample concerns a Chord- network and a node Ni 
which leaves the network by executing the FairLeave rule. Then, Ni tries to trans- 
fer its key value-table to N successor uy However, suppose that N successor ^ leaves 
the network in the same moment. Obviously, the content of JVj's keyvalue-table 
will be lost. 

Several improvements of Chord protocol are proposed to increase robustness: 

• in 2D| each Chord node can maintain a list containing the node's first r 
successors, and 

• it is usual in software implementations |10j to make multiple copies of all 
(key,value)-j>SLiTS (for example, by different hash functions). 

Note that those changes might improve performance of a Chord network, but do 
not affect correctness. Quite simple generalizations of Example 15.11 would produce 
incorrect behavior. 

The Chord protocol is introduced in [20l422| . The papers analyze the protocol, 
its performance and robustness, under the assumption that the nodes and keys 
are randomly chosen and give several theorems that involve the phrase with high 
probability, for example: "With high probability, the number of nodes that must be 
contacted to find a successor in a iV-node network is O(logiV)" [22, Theorem IV. 2]. 
Our Theorem 15.11 corresponds to that statement, but only proves correctness of 
the rule FindSuccesor. Theorem [22j Theorem IV.2] considers the so-called pure 
join model of Chord which does not allow (un)fair-leaving, while we permit failures 
of nodes, but restrict runs to be regular. If we assume the random distribution, 
it is easy to see that the same complexity result can be obtained. On the other 
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hand, in the theorems [22j Theorem IV. 5] and [22] Theorem IV. 6] behavior of 
FindSuccesor is analyzed when failures of nodes can happen. Then, to prove high 
probability of correctness it is necessary to involve the aforementioned improvement 
of Chord which concerns lists of successors of nodes. The only statement in [22] 
which avoids the mentioned phrase about high probability is Theorem IV. 3. It 
(corresponds to our Theorem 15.31 and) proves that inconsistent states produced by 
executing several concurrent Join-rules are transient, i.e., that after the last Join- 
rule, nodes in a network will form a cycle. That theorem also considers the pure 
join model. Although in this paper we consider regular runs, since in Theorem 
15.31 (un)fair-leaving is not allowed, the theorem holds for all runs. More general 
sequences of concurrent joining and leaving are considered in [17], where a lower 
bound of the rate at which nodes need to maintain the system such that it works 
correctly is given (again) with high probability. In the previous sense, our Corollary 
15.31 corresponds to the main statement jTTJ Theorem 5.9] of that paper. 

It is not quite clear how to compare these two approaches, but in our opinion 
there is benefit from both of them. One can argue that the probabilistic approach 
(providing lower bounds of probabilities) is useful to study robustness of protocols. 
On the other hand, we are able to describe sequences of actions leading to (un) stable 
states of Chord networks, which can be useful when one analyzes properties of 
systems that incorporate Chord and assume its correctness, as it is the case with 
non-relational database systems. 

In [15] the theory of stochastic processes is used to estimate the probability 
that a Chord network is in a particular state. In [TJE] Chord's stabilization algo- 
rithm is modelled using the 7r-calculus and it's correctness is established by proving 
the equivalence of the corresponding specification and implementation. Possible 
departures of nodes from a network are not examined in this approach. The pa- 
per |25] presents the results of testing of Chord protocol which is based on random 
simulations. It reports some failures and proposes modifications in the Join and 
Stabilization rules. For example, an update of the successor of a node is added to 
the Stabilize-rule to decrease the number of steps needed to obtain a stable state. 
Our specification of Chord includes that modification. In [26,27 the Alloy formal 
language is used to prove correctness of the pure join model. The same formaliza- 
tion produces several counterexamples to correctness of Chord ring-maintenance. 
In our approach, since we consider the regular runs only, those examples do not 
cause incorrectness. However, as we argue above, our results do not imply that in 
a more general framework some shortages cannot happen. 

Finally, we note that the mentioned papers mainly consider maintaining of topo- 
logical structure of Chord networks, and do not analyze handling of (key, value)- 
pairs presented in our Theorem 15 . 71 and Corollary [53] 



7. Conclusion 

In this paper we have presented an ASM-based formalization of the Chord pro- 
tocol. We have proved that the proposed formalization is correct with respect to 
the regular runs. Up to our knowledge, it is the first comprehensive formal anal- 
ysis of Chord presented in the literature which concerns both maintenance of the 
ring topology and data distribution. We have also indicated that if we consider all 
possible runs, incorrect behavior of Chord protocol could appear. 
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Possible direction for further work is to apply similar technique to describe other 
DHT protocols. For example, an interesting candidate for examination in the ASM- 
framework could be Synapse, a protocol for information retrieval over the inter- 
connection of heterogeneous overlay networks defined in |18j , and applied in |19j . 

Another challenge could be verification of the given description in one of the 
formal proof assistants (e.g., Coq, Isabelle/HOL). It might also produce a certified 
program implementation from the proof of correctness of our ASM-based specifica- 
tion. 
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Appendix A. Appendix: Chord Rules - Low Level Description 

We give here a detailed specification of the module and rules introduced in 
Section l4~2l 

Peer_agent Module= 

/ / Peer tries to start/connect to the Chord network 
if mode(M e) = notjconnected then 
if mode_join(Me) = undef then 
if action = undef then 

/ / Peer has been not active and can choose to skip or to join 
choose a in Join 

action := a 
endchoose 
else 

if action = join then 
seq 

/ / Checking the list of known nodes of Chord network 
known := known jnodes{M e) 
if known — undef then 

/ / Start new Chord network 

Start 
else 

/ / Join to existing Chord network 
Join — 

if mode -j oin(M e) = undef then 
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JoinBegin 
else 

/ / Continuing joining process depending on received messages 
if mode-join(Me) = wait.f or successor then 

JoinGetSuccessor 
else 

if mode-join(Me) = wait _for _keys then 

JoinGetKeys 
else 

par 

/ / Connection was successful - start stabilization pro- 
cess 

if id(Me) ^ undef then 
mode(Me) := connected 
1 1 Connection was not successful - try again 
else 

mode(Me) :— not ..connected 
endif 

mode-join(Me) = undef 
action := undef 
endpar 
endif 
endif 
endif 
endif 
endseq 
else 

/ / the skip action has been chosen 
action := undef 
endif 
endif 

/ / Connected node chooses what to do 
else 

if mode(Me) = connected then 

/ / Checking if id(Mc) can communicate with other nodes 
if ping (id(M e)) — true then 
par 

/ / Read all new messages 

ReadMessages 

/ / Begin new stabilization cicle 

Stabilize = 

if modestabilize(Me) — undef 

A(action ^ fair leave V action ^ unfair leave) then 
StabilizeBegin 
else 
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if modestabilize(Me) = wait-f or .predecessor then 

StabilizeWithPredecessor 
else 

if modestabilize(Me) = wait-f or-keys then 

StabilizeSetKeys 
else 

if modestabilize(Me) — wait-f or successor then 

StabilizeUpdateSuccessor 
else 

if modestabilize(Me) = wait-f or successor -keys the 

StabilizeUpdateKeys 
endif 
endif 
endif 
endif 
endif 

/ / Cheking connectivity of the predecessor 

UpdatePredecessor 

/ / Update value for a ringer table member 

UpdateFingers = 

if mode-finger s{Me) = undef 

A(action ^ fair -leave V action ^ unfair leave) then 
UpdateFingersBegin 
else 

if mode-fingers(Me) = wait-f or -response 

UpdateFingersSet 
endif 
endif 

/ / Allow other actions beside Join 

ExtendedJoinModel = 

seq 

if action — undef then 

choose a in Action 
action := a 

endchoose 
endif 
par 

/ / Allow leaving 
Leaving Actions = 
par 

FairLeave = 

i f action = fair leave A modestabilize(Me) = undef 
Amode-fingers(Me) = undef then 
/ / Node is leaving network in a fair way 
if modeleave(Me) = undef then 
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FairLeaveGetSuccessor 
else 

if mode Jeave(M e) = wait_f or. successor then 

FairLeaveUpdateSuccessor 
else 

par 

FairLeaveExchange 
mode(Me) := not -connected 
action := undef 
endpar 
cndif 
endif 
endif 

i f action = unfair leave A modestabilize(Me) — undef 
Amode.f inger s(M e) = undef then 
par 

/ / Node is leaving network in a unfair way 
UnfairLeave 
mode(Me) := not.connected 
action := undef 
endpar 
endif 
endpar 

/ / Allow key / value handling 
KeyValueHandling = 
par 

Put = 

if action = put then 

/ / Inserting new (key, value) pair 
if mode .put (M e) = undef then 
seq 

(key , value) := key.value(Me) 

PutFindResponsible 
endseq 
else 
par 

PutKeyValue 
action := undef 
endpar 
cndif 
cndif 
Get = 

if action = get then 

if mode-get(Me) = undef then 
seq 
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/ / Searching for existing value 
key := keys(Me) 
GetKey 
endseq 
else 

if mode-get(Me) = wait_for_key then 

GetValue 
else 
par 

GetFinish 
action := undef 
endpar 
endif 
endif 
endif 
endpar 
endpar 
endseq 
endpar 
else 

/ / Connection problems detected - reset connection 
par 

mode(Me) := not -connected 
mode_stabilize(M e) := undef 
mode-fingers(Me) := undef 
modeJeave(Me) := undef 
mode_put(Me) := undef 
mode-get(Me) := undef 
action :— undef 
endpar 
endif 
endif 
endif 

Start= 
seq 

/ / Setting id(Me) according to the hash function 

id(Me) := hash(Me) 

1 1 Set all messages to id(Me) to empty 

ClearMessages 

/ / Initialization of local structure 

par 

predecessor (id(M e)) := undef 
successor (id(M e)) := id(Me) 
finger(id(Me)) := [] 
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next(id(Me)) := -1 
keyvalue{id{Me)) := [] 
endpar 
endseq 

JoinBegin= 

seq 

/ / Setting id(Me) according to the hash function 
id(Me) := hash(Me) 
//If connection is succesful 
if id(Me) ^ undef then 
seq 

/ / Set all messages to id(Me) to empty 
ClearMessages 
/ / Initialize local structure 
seq 
par 

predecessor ■(id(Me)) :— undef 
finger(id(Me)) := [] 
next(id(Me)) := -1 
endpar 

/ / Ask for the successor 

communication(id(Me), known) .add((findsuccessor, (id(Me), id(Me)))) 
mode -join (Me) :— wait-f or successor 
endseq 
endseq 
else 

/ / Connection was not succesful 
mode-join(Me) := finished 
endif 
endseq 

JoinGetSuccessor= 

/ / Get the message with the information on the successor 
forall m g Message with m — communication(sender,id(Me)) 
if m = (successor -found, content) then 
if content = (id(Me), successor) then 
par 

/ / Set successor 

successor ( id (Me) ) :— successor 
/ / Ask successor for keys 
communication(id(Me), successor (id(Me)) 

.add((request -and -remove -key value, undef) ) 
mode-join(Me) :— wait-for-keys 
1 1 Remove processed message 
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communication{sender, id(M e)) .remove(m) 
endpar 
endif 
cndif 
endforall 

JoinGetKeys= 

/ / Get the message with keys and values 

forall m £ Message with m = communication(sender,id(Mej) 
if m = {received-key value, content) then 
par 

/ / Set keys 

keyvalue(id(M e) := content 
mode.join(Me) := finished 
1 1 Remove processed message 
communication(sender, id(Me)).remove(m) 
endpar 
endif 
endforall 

FairLeaveUpdateSuccessor= 
/ / Check the successor and update it if necessary 
if -iping(successor(id(Me))) then 
seq 

communication(id(Me), id(Me)).add((findsuccessor, (id(Me)©jvl, id(Me)))) 
modeJeave(Me) := wait. for. successor 
endseq 
else 

modeJeave(Me) := proceed Jo. finish 
endif 

FairLeaveGetSuccessor= 

/ / Get the message with the information on the successor 
forall m £ Message with m = communication{sender,id{Me)) 
if m = (successor .found, content) then 
if content = (id(Me), successor) then 
par 

/ / Set successor 

successor ( id (Me) ) := successor 
modeJeave(Me) := proceed do .finish 
1 1 Remove processed message 
communication(sender, id(Me)).remove(m) 
endpar 
endif 
endif 
endforall 
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FairLeaveExchange= 
seq 

//If Me is not the last node 

if ^(successor (id(Me)) = predecessor(id(Me))Apredecessor(id(Me)) = id(Me)) then 
par 

// Send keys/values to the successor 

communication(id(Me), successor (id(Me)).add((send.keys, keyvalue(id(M e)))) 
1 1 Exchange pointers between the successor and the predecessor 
communication(id(Me),successor(id(Me)) 

.add{ (send.predecessor, predecessor (id(Me)))) 
communication(id(M e) , predecessor (id(M e)) 
.add((sendsuccessor, successor (id(Me)))) 
1 1 Clean local memory 
predecessor {id{M e)) := undef 
successor (id(M e)) := undef 
finger{id(Me)) := [} 
keyvalue(id(Me)) := [] 
endpar 
endif 
endseq 

StabilizeBegin= 

/ / Check if id(Me) can communicate with other nodes 
if ping (id(M e)) = true then 

/ / Check the successor's connection 

if ping(successor(id(Me))) = true then 
par 

/ / Get information on the predecessor of the successor 
communication(id(M e) , successor (id(M e))) .add( (get_predecessor, undef)) 
mode_stabilize(Me) :— wait-f or. predecessor 

endpar 
c 1 s e 

par 

/ / Find new successor 

communication(id(Me) 7 id(Me)).add({findsuccessor, (id(Me)©Arl, id(Me)))) 
modestabilize(Me) :— wait_f or. successor 
endpar 
endif 
endif 

StabilizeWithPredecessor= 

/ / Get the message with the information on the predecessor of the successor 
forall m G Message with m = communication(sender,id(Me)) do 
if m = (received-predecessor, content) then 
seq 
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x := content 

1 1 Remove processed message 
communication(sender, id(Me)).remove(m) 
J I There is a new node between id(Me) and the successor 
if (1/ undef A member .of(x, id(Me), successor (id(Me)))) then 
par 

// Update the successor and take keys/values from it 
successor (id(M e)) := x 
communication(id(Me), successor (id(Me)) 

.add{ (request-andjremove-keyvalue, undef) ) 
modestabilize(Me) :— wait -for -keys 
endpar 
cndif 

/ / id(Me) is a new node between successor and its predecessor 

if x — undef\/(x^undefAmemberjof(id(Me),x,successor(id(Me)))) then 

/ / Notify the successor to change the predecessor 

par 

communication(id{Me) , successor (id(Me) ) ) .add( {set predecessor, undef) ) 
modestabilize(Me) :— undef 
endpar 
endif 
endseq 
endif 
endforall 

StabilizeSetKeys= 

/ / Get the message with keys and values 

forall toG Message with m = communication(sender,id(Me)) 
if m = (received-keyvalue, content) then 
par 

/ / Add keys/values to local table 
keyvalue(id(Me) .add(content) 
modestabile(Me) := undef 
1 1 Remove processed message 
communication(sender, id(Me)).remove{m) 
endpar 
endif 
endforall 

StabilizeWaitSuccessor= 

// Get the message with the information on the successor 
forall tog Message with m — communication(sender,id(Me)) 
if m = (successor-found, content) then 

if content = (id(Me) (Bn 1, successor) then 
par 
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// Set successor 
successor ( id (Me) ) := successor 
// Ask successor for keys 
communication{id(M e) , successor (id(M e)) 

.add({request-and-remove-key value, undef)) 
modestabilize(Me) := wait_f or .successor _keys 
1 1 Remove processed message 
communication{sender, id(M e)) .remove(m) 
endpar 
cndif 
endif 
endforall 

StabilizeUpdateKeys= 

/ / Get the message with keys and values 

forall m G Message with m — communication(sender,id(Me)) 
if m = (received-key value, content) then 
par 

/ / Add keys/values to local table 
keyvalue(id(M e) .add(content) 
modestabilize(Me) :— undef 
1 1 Remove processed message 
communication(sender, id(Me)).remove(m) 
endpar 
endif 
endforall 

UpdatePredecessor= 

/ / Check if id(Me) can communicate with other nodes 
if ping(id(Me)) — true then 

/ / Check the predecessor's connection 

if ping(predecessor(id(Me))) ^ true then 
predecessor(id(Me)) :— undef 

endif 
endif 

UpdateFingersBegin= 

/ / Update a finger table item 

par 

next(id(Me)) := (next(id(Me)) (B M 1) + 1 

communication(id(Me), id(Me)).add((find_successor, (id(Me)® N 2 next( - ld{ - Me ^~ 1 , id(Me)))) 
mode_fingers(Me) :— wait_f or .response 
endpar 

UpdateFingersSet= 

/ / Get the message with the information on the successor 
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forall tog Message with to = communication(sender,id(Me)) 
if m = {successor -found, content) then 

if content = (id(Me) ® N 2™ ea: *( id ( Me ))- 1 , successor) then 
par 

/ / Set current finger table item 

finger (id(Me)).listitem(next(id(Me))) := successor 
1 1 Remove processed message 
communication(sender, id{M e)) .remove(m) 
1 1 Begin update next table item 
mode-fingers(Me) := undef 
endpar 
endif 
endif 
endforall 

PutFindResponsible= 

/ / Insert new (key, value) pair into Chord network 
par 

/ / Find the responsible node for hash(key) 

communication(id(Me), id(Me)).add((findsuccessor, (hash(key) , id(M e)))) 
mode-put(Me) := wait-f 'or -responsible 
endpar 

PutKey Value= 

/ / Get the message with the information on the responsible node 
forall to G Message with m — communication(sender,id(Me)) 
if m = (successor -found, content) then 
if content = (hash(key), successor) then 
par 

x := successor 

if x ^ undef then 

/ / Send (key, value) to the responsible node 

communication(id(Me), x).add((sendJkeys, [(hash(key), value)])) 
else 

Skip 
endif 

/ / Remove processed message 
communication(sender, id(Me)).remove(m) 
1 1 Finishing this instering operation 
mode-put(Me) := undef 
endpar 
endif 
endif 
endforall 
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GetKey= 

/ / Begin search for given value 
par 

communication(id(Me), id(Me)) .add((findsuccessor, (hash(key),id(Me)))) 
mode.get(Me) := wait-forJzey 
endpar 

GetValue= 

/ / Get the message with the information on the responsible node 
forall me Message with m — communication(sender,id(Me)) 
if m = (successor _ found, content) then 
if content = (hash(key), successor) then 
par 

/ / Ask responsible node for the value connected with the given key 
responsible := successor 

communication(id(Me), responsible) .add((get, hash(key))) 
1 1 Remove processed message 
communication{sender, id(M e)) .remove(m) 
mode-get(Me) := wait.f or. value 
endpar 
endif 
endif 
endforall 

GetFinish= 

/ / Get the message with the information on the asked value 
forall m E Message with m — communication(sender,id(Me)) 
if m = (value-found, content) then 

if content = (hash(key) , successor) then 
par 

value := content 
1 1 Remove processed message 
communication(sender, id(Me)).remove(m) 
mode-get(Me) := undef 
endpar 
endif 
endif 
endforall 

Search= 

/ / Find given value in local key /value table 
seq 

choose value in Value satisfying 
(hash(key) , value) G keyvalue(id(Me)) 
content :— value 
endchoose 
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communication(id(M e) , sender). add(value -found, content) 
endseq 

ClearMessages= 

/ / Clear messages remaining by the node which had same id 
f o r a 1 1 node G Chord 

communication(node,id(Me)) := [] 
endforall 

ReadMessages= 

/ / Process received messages 

forall to e Message with m — communication(sender,id(Me)) do 
seq 
par 

/ / Send proper keys/values to new predecessor 
if to = (request-and-remove_key value, content) then 
seq 

res := | 

forall (h,v) <G keyvalue(id(Me)) with h ^ sender do 
par 

keyvalue(id(Me)).remove((h, v)) 
res.add((h, v)) 
endpar 
endforall 

communication(id(Me) 7 sender) :— {received-key value, res) 
endseq 
endif 

/ / Add received keys/ values to local table 
if to = (send-keys, content) then 

keyvalue{id(M e)) .add{content) 
endif 

/ / Set given predecessor 

if to = (set -predecessor, content) then 

predecessor {id(Me)) := sender 
endif 

/ / Send information on local predecessor 
if to = (get -predecessor, content) then 

communication(id(Me), sender) := (received-predecessor, predecessor(id(Me))) 
endif 

/ / Process query on respoinsible node 
if to — (findsuccessor, content) then 

FindSuccessor 
endif 

/ / Process search on given value in local table 
if to = (get, content) then 
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Search 
e n d i f 
endpar 

/ / Remove processed message 
communication(sender, id(M e)) .remove(m) 
endseq 
e n d f o r a 1 1 

FindSuccessor= 
seq 

(h, starter) := content 

II If h is between id(Me) and its successor responsible node is the successor 
i f ping(successor(id(Me)))/\member_of(h, id(Me), successor (id(M e))) then 

communication(id(M e) , starter) .add((successor _/ ound, (h, successor (id(M e))))) 
else 

/ / Find the closest preceding node in local finger table 
seq 

index :— undef 

choose i in {1,...,M} satisfying 

member jof (finger (id(M e)) .listitem(i) , id(M e) , h) 

A^member_of(finger(id(Me)).listitem(i + l),id(Me), h))) 
Aping(finger(id(Me)).listitem(i)) = true 
II h is between two values in local finger table 
index := i 

II h is bigger then all values in local finger table - the closest is the 
maximal element 

if index = undef then 

choose i in {1,...,M} satisfying 
ping( finger (id(M e)) .listitem(i)) = true 

A(Vj e {2, . . . ,M + l\j > i})ping(j) = false 
index := i 
endif 

/ / Forward query to it 

communication(id(M e) , finger (id(M e)) .listitem(index)) 
.add((find_successor, (h, starter))) 
endseq 
endif 
endseq 
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